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Introduction: The context 

The current focus in higher education on quality, and the current 
interest in using performance indicators are reflections of the 
context in which higher education is operating. 

That context can be characterised in terms of a period of signifi¬ 
cant change and adjustment. Universities have been challenged by 
a series of dilemmas. It has been a period of significantly increased 
participation in the final year of secondary education leading to 
increased demand for higher education from school leavers. In 
parallel there has been the adoption of more flexible entry provisions 
to address issues of equity and access, leading to increased demand 
for access to higher education from groups other than school leavers. 4V 
And the prevailing economic conditions have reduced the availabil¬ 
ity of alternatives to higher education. They have also changed 
patterns of demand for courses. The current downturn in demand for 
Education courses is but one example. 

This increase in demand and the resulting increase in higher 
education places has occurred in an environment of general eco¬ 
nomic constraints, reducing resources and increasing competition 
for scarce resources. Partly as a result of these resource constraints, 
and partly also because of the perceived slowness of change in 
universities, there have been persistent and increasing calls for 
improved efficiency and public accountability in all aspects of 
higher education. 

This has been reflected in pressure for the development of more 
objective and systematic procedures for the evaluation of universi¬ 
ties, and for systematic monitoring of the performance of universi¬ 
ties and the higher education system as a whole. 

There has been accelerating change toward more formal, routine 
and quantitative approaches to evaluation of higher education. The 
focus has been on monitoring performance and productivity to assist 
institutions to improve their efficiency and accountability. 

Some key developments 

It is important to acknowledge some of the recent milestones 
which have influenced the way in which performance indicators are 
currently viewed. 

1. The Review of Efficiency and Effectiveness in Higher Educa¬ 
tion (CTEC, 1986) marked a significant advance in government 
pressure for visible progress in establishing systematic mechanisms 
for performance appraisal of higher education institutions. To that 
time such performance appraisal had been largely at the discretion 
of the institution, intermittent, predominantly subjective, confined 
essentially to single institutions with inadequately defined central 
data systems giving little basis for comparability. 

This Review encouraged the further development and application 
of performance indicators both within institutions and across the 
higher education system to assist in identifying strengths and 
weaknesses and in measuring progress toward specified institutional 
and systemic objectives. 

2. The Discipline Review of Engineering {Williams 1988) which 
reported in 1988 systematically attempted to define and apply a 
range of input and performance indicators for higher education 
institutions, which to some degree covered the major functions of 


higher education in professional discipline areas, 

3. The Green Paper (Commonwealth of Australia 1987) marked a 
strengthening of government policy on institutional evaluation and 
performance appraisal. That paper proposed in specific terms that 
institutional performance in its various forms be quantitatively 
assessed and that this assessment should have some influence on 
institutional funding. The paper explicitly stated the Government’s 
intention ‘to fund on output and performance’. Importantly, there 
was an acceptance that performance indicators needed to be accept¬ 
able to both institutions and the Government. 

The Government expects institutions, as part of their strategic 
planning , to give consideration to indicators that would help in 
measuring the achievement of their goals. In time, such indicators 
that are agreed to be useful would be built into the general funding 
system. 

The need for performance indicators acceptable to both institutions 
and the Government points to the likelihood of further demands on 
the higher education statistical information base. (Commonwealth 
of Australia 1987, p.42) 

4. The AVCC/ACDP Working Party on Performance Indicators 
was jointly established by AVCC and ACDP in response to these 
proposals in the Green Paper. This joint working party attempted to 
define what they saw as ‘a proper context’ in which performance 
indicators should be used. 

The joint working party sounded some warnings to those tempted 
to enthusiastically embrace performance indicators: 

Something resembling a Cargo Cult seems to have grown up around 
the notion of performance indicators, so that all manner of powers 
and virtues are ascribed to them and expectations are aroused that 
by collecting and using them great benefits will miraculously 
result. ( AVCC/ACDP 1988, p.l) 

In their report of December 1988, they proposed that ‘a proper 
context’ for the use of performance indicators be established by a 
policy agreed by institutions: 

the best manner of assessing performance is by the judgement of 
knowledgeable and independent people, and that 

an agreed set of indicators will form part of the material available 
to those conducting the reviews. (A VCC/ACDP 1988, p.4) 

They went on to try to identify a range of possible indicators which 
would be acceptable to both institutions and the Government. They 
proposed that, in selecting performance indicators among the things 
that should be kept in mind were: 

- indicators should be clearly related to the institution’s prime 
functions and objectives 

° indicators should cover as many of the institution’s prime 
functions and objectives as possible 

• indicators should form a coherent set 

* indicators are of greater significance when considered in groups 
than singly 

s as few indicators as possible should be used 
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• indicators should be valid, reliable and verifiable 

9 indicators should be defined and collected in a uniform and 
agreed fashion 

9 trends in indicators over time are likely to be more significant 
than comparisons at a particular time. 

The Government welcomed this initiative and in the White Paper 
encouraged the development of performance indicators along the 
lines proposed by the Joint Working Party. 

5. The White Paper (Commonwealth of Australia 1988) stated 
The Government supports the development of a funding system that 
responds to institutional performance and the achievement of 
mutually agreed goals. It intends to develop funding arrangements 
that take into account a range of output, quality and performance 
measures... 

It also stated 

As soon as practicable, indicators which are agreed to be useful and 
appropriate will be incorporated into the Commonwealth's general 
funding arrangements for higher education. 

It also gave some indication of the scope of performance indica¬ 
tors it had in mind: 

The range of indicators to be developed should cover such issues as 
student demand and course completion rates, quality of teaching 
and curriculum design, relative staffing provision and measures of 
academic staff performance in various aspects of research, publi¬ 
cation, consultancy and other professional services. Indicators of 
performance against equity goals and measures of organisational 
efficiency should also be included in this process. (Commonwealth 
of Australia 1988, p.86) 

6. The Trial Evaluation Study of Performance Indicators in Higher 
Education (Linke 1991), funded by DEET, investigated a range of 
indicators, and recommended definitions of such indicators based on 
their analysis. In addition, the Research Team recommended, inter 
alia: 

• the collection and analysis of data required to generate the 
proposed range of context, performance and social equity 
indicators be undertaken. 

• the annual publication of summary data on these indicators, 
with appropriate explanatory details. 

• further research and development of performance indicators in 
higher education. 

• studies on the links between educational input and output 
measures. 

In its report it distinguished: 

Indicators of Institutional Context 
Performance Indicators - Teaching and Learning 
Performance Indicators - Research and Professional Services 
Participation and Social Equity Indicators 
It recommended that further work needed to be undertaken in the 
development of performance indicators in two key areas in which the 
performance of universities should be monitored, namely research, 
and equity and participation. 

DEET has funded subsequent studies in these areas. 

7. The Higher Education Council report on Higher Education: 
Achieving Quality (Higher Education Council 1992) recommended 
a national structure to invite universities to participate in a regular 
review and audit of their mechanism for monitoring and improving 
the quality of their outcomes. 

8. The Government responded by setting up a Committee for 
Quality Assurance in Higher Education, with the following Terms of 
Reference: 

provide public advice on the condition of quality in the higher 
education system. In doing this it may... 


1. Invite universities to participate in a regular review and audit of 
their mechanisms for monitoring and improving the quality of their 
outcomes, and: 

(a) examine portfolios volunteered by universities showing what 
they have put in place to assure and improve quality; 

(b) evaluate how they have assessed the effects of their policies and 
processes, including summaries of their own assessments of their 
performance; 

(c) conduct interviews and visits as appropriate; and, 

(d) use existing, nationally-based, data. 

2. Recommend direct to the Minister on the allocation of the 
specially designated funds to universities to recognise achieve¬ 
ments demonstrated by the effectiveness of policies and procedures 
as demonstrated through an evaluation of their assessment of the 
quality of their outcomes. 

With an annual allocation of $76 million in prospect, institutions 
found the invitation irresistible. Nothing focuses the mind as much 
as the prospect of additional funding, and the fear of being excluded 
from the designated 'Quality Club’. 

The result has been the expenditure of a great amount of effort as 
institutions attempt to collate data which will convince the commit¬ 
tee that they can demonstrate quality outcomes, and therefore should 
be doubly rewarded by being designated as part of the in-group of 
“quality institutions” and by additional funding. 

Under what conditions can performance indicators measure qual¬ 
ity of outcomes of higher education? 

This is not a simple question. I will attempt to answer it by 
focusing on a series of Key Issues. 

KEY ISSUE 1: Performance areas sampled by indicators 

It is essential that the performance indicators used to measure the 
quality of a university’s performance should be related to the 
university’s prime functions and objectives. It is also essential that 
the set of indicators used should be an adequate and representative 
sample of the whole set of the university’s prime functions and 
objectives. 

In this regard, a set of performance indicators is like an end-of- 
course examination, which must also adequately sample the whole 
set of the course objectives. 

The set of performance indicators must not focus on a few 
unrepresentative areas, nor should it focus on areas in which data is 
more readily available in preference to those in which data is 
relatively difficult to gather. It must also be recognised that the same 
set of performance indicators may not be equally appropriate in all 
areas of a university’s operations. 

It is also important that performance indicators directly measure 
the performance concerned, and that the infidelity of indirect meas¬ 
urement is minimised or avoided. 

A related issue arises if an external agency defines the set of 
performance indicators to be used. This situation has the potential to 
divert the institution’s functional objectives toward those valued and 
measured by those externally imposed performance indicators. This 
is particularly the case if that external agency is perceived to be 
powerful, influential or likely to reward performance as measured on 
those indicators. 

KEY ISSUE 2: Purpose for using performance indicators 

It is important to know the purpose for using performance indica¬ 
tors before deciding which indicators are appropriate, and how they 
should be used. If the purpose is essentially formative (i.e. institu¬ 
tional development), a different approach may be appropriate than 
for summative evaluation of an institution’s performance, or for 
differential funding allocation purposes. For institutional develop¬ 
ment purposes, the focus is on identifying areas where improvement 
in performance is required. By contrast for summative evaluation 


purposes the focus is on accurately demonstrating institutional 
achievements in key areas. 

KEY ISSUE 3: Comparative basis for interpretation of the 
data 

It is important to distinguish between three bases for interpreting 
data gathered on performance indicators. In norm-referenced inter¬ 
pretations, a university’s performance is compared with that of other 
universities to demonstrate relative performance. Such interpreta¬ 
tions lead to judgements such as the university’s performance being 
above or below average, the best or the worst etc. The focus is not 
how good the performance is, but how it compares with that of 
others. 

In criterion-referenced interpretations, performance is compared 
with a pre-defined criterion or standard for performance, leading to 
the judgement of whether or not the desired standard has been 
achieved. 

In self-referenced interpretations, trends in a university’s per¬ 
formance over time are examined with a view to demonstrating 
improvement in performance in desired areas. 

Each of these three types of interpretation has its place, and a 
contribution to make to understanding performance. 

KEY ISSUE 4: Level of aggregation of data 

The trial evaluation study (Linke 1991) demonstrated that various 
levels of aggregation are appropriate for different performance ? > 
indicators if they are to be meaningful. These include 

• institution 

• AOU 

• department 

• award 

• discipline 

• Fieid of Study 

For each performance indicator, an appropriate level of aggrega¬ 
tion and reporting of data needs to be determined. There is a fine 
balance to be struck. Inappropriate aggregation of data which is not 
comparable across the groups being aggregated will lead to a 
reduction in meaning. On the other hand, excessive division of data 
into minute groupings tends to lead to excessive amounts of data and 
to data of low reliability because it is based on very small numbers. 

KEY ISSUE 5: Quality of basic data 

Many of the applications of performance indicators have been 
based on data compiled and collected for purposes that are essen¬ 
tially different. Such data must be used cautiously for purposes other 
than that for which it was originally collected. Often the data 
contains errors, and can be outright misleading. It is a good rule to 
check the validity and accuracy of the data used before that data is 
used to represent or to interpret performance outcomes. 

KEY ISSUE 6: Efficiency of data versus triangulation 

Performance indicators are of far greater significance when con¬ 
sidered as groups or constellations than when considered singly. The 


attraction of developing a small set of indicators which efficiently 
measure a range of outcomes must be tempered with the increased 
validity likely to be produced by triangulation of data from several 
related indicators. It must be acknowledged that many performance 
indicators will be indirect measures of the target performance. 
Under such circumstances, triangulating performance from several 
indirect measures is a prudent approach. 

KEY ISSUE 7; Relationship between input and output 
variables 

institutional context variables such as staff-student ratios, quality 
of student intake and resource richness are likely to be related to 
performance outcomes. Accordingly, performance outcomes cannot 
be considered in isolation from a range of contextual inputs. It is 
therefore inappropriate to report outcome measures in isolation from 
measures of the institutional context in which those outcomes are 
produced. 

KEY ISSUE 8: Do current sets of performance indicators 
provide an adequate basis for funding allocation? 

Before performance indicators can be used for this purpose, both 
universities and governments must be confident that sets of perform¬ 
ance indicators have been developed which accurately, reliably and 
validly measure the full range of prime purposes and objectives of 
universities. They must also be sensitive enough to reflect the 
increasing diversity in purposes of different universities as they 
shape their post-merger missions. 

In my opinion that situation has not yet been reached, although 
impressive progress has been made in the last decade, and continues 
to be made. 
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